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TITLE OF THE INVENTION 
IMAGE PROCESSING APPARATUS AND METHOD 
5 BACKGROUND OF THE INVENTION 

Field of the Invention 

[0001] The present invention relates to an image 
processing apparatus and method, and specifically, to 
10 interframe coding on video image data and decoding on coded 

video image data. 
Description of the Related Art 

[0002] Intraframe coding methods, such as Motion JPEG 
(Joint Photographic Experts Group) and DV (Digital Video), 
15 and interframe predictive coding methods, such as H.261, 

H.263, MPEG-1, and MPEG-2, are known image coding methods of 
the related art. 

[0003] These coding methods have been internationally 
standardized by the International Organization for 
20 Standardization (ISO) and the International 

Telecommunication Union (ITU). 

[0004] Intraframe coding methods are adequate for 
apparatuses requiring video image editing and special 
playback since the methods perform separate coding in units 
25 of frames and therefore, frames can be easily managed. Also, 
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Interframe coding methods have a feature of high coding 
efficiency since the methods use interframe prediction. 
[0005] In addition, the coding standards include a new 
coding standard called "MPEG- 4" which is a next -generation 
5 general-purpose multimedia coding standard that can be used 

in many fields of computers, broadcasting, and communication, 
etc. 

[0006] In the H.261, H.263, MPEG-1, and MPEG-2 standards, 
the frame-unit coding in use is divided into three types, 

10 that is, an intra-picture (I picture) that is only 

intraframe-coded, a predictive -coded picture (P picture) 
that also performs interframe prediction from the closest 
past frame, and a bidirectionally- coded picture (B picture) 
that additionally performs interframe prediction from the 

15 closest future frame. 

[0007] In the interframe coding, all frames must be 
sequentially transmitted for referring to the closest past 
frame. When one party performs data transmission after 
establishing a communication link with another party in the 

20 case of, for example, a telephone line or an ISDN 

(Integrated Services Digital Network) line, no problem 
occurs since data sequentially reaches the other party 
without becoming lost on the path between both parties. 
However, since, in the case of, for example, a local area 

25 network (LAN) and an asynchronous transfer mode (ATM) , no 
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communication link Is established, and coded data Is 
transmitted In a form divided Into smaller data units 
(packets or cells), some packets may be lost In the 
communication channel, and the order of packets may be 
5 switched due to the use of different communication channels. 

[0008] In order that the packet receiving party may know 
the original order of packets, even If the order of packets 
Is switched, the reliability of the network Is enhanced such 
that the packet transmitting party transmits serially 

10 numbered packets, or a protocol (e.g., TCP (Transmission 

Control Protocol)) which confirms the arrival of packets or 
which sends back an undelivered- packet re-send request from 
the receiving party to the transmitting party. 
[0009] When the operation of the network Is unstable 

15 causing packets to be frequently lost, the user of the above 
protocol to perform re- sending accumulates transmission 
delays. Thus, the above protocol Is not suitable for real- 
time transmission of video Images. 

[0010] In addition. In broadcast and multicast which have 
20 become popular as use of the Internet has become widespread, 

and which Is used for multipoint data transmission, a 
mechanism for transmitting data to a plurality of points by 
performing packet transmission one time Is used. In this 
mechanism, when a packet Is lost during packet transmission 
25 to one point, re -sending such as using the above protocol 
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greatly Increases the network load because, despite normal 
arrival of the first packet, the other points must re-send 
identical packets through the network. Accordingly, in 
broadcast and multicast, it is common to use a protocol 
5 (e.g., UDP (User Data Protocol)) that does not perform re- 

sending. Nevertheless, the use of this protocol increases a 
probability that lost packets occur. 

[0011] When a wireless network is used, not only the case 
of transmitting packets obtained by dividing data, but also 

10 even the case of transmitting data after establishing a link 
tends to increase the data error rate or the lost 
packet/data rate. In particular, if the receiving party 
detects an error, when a received signal includes errors 
beyond its error correcting ability, a method that abandons 

15 data in a certain section in order to perform normal 
processing on the other parts of data is employed. 
Accordingly, the amount of lost data is larger than that in 
the case of a wired network. 

[0012] By using video image data as an example of lost 
20 data in packet transmission, a specific example in an MPEG- 4 

case is described below. 

[0013] Fig. 1 is an illustration of an example of lost 
frames in packet transmission of video image data. 
[0014] Fig. 1 shows MPEG- 4 video -image -data frames a to e. 
25 Frame a is an I frame that is in traframe- coded. Frame b to 
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frame e are P frames that are interf rame-predictive-coded. 
[0015] As shown in Fig. 1, when frame c is lost during 
transmission, and when freune c cannot be decoded due to a 
delay in decoding on frcune c, frame c cannot be decoded 
5 until the next I frame i (not shown) arrives, so that it is 

impossible to decode frames d, e, which are P frames 

existing until the next I frame i arrives. 
[0016] Accordingly, in order to ensure transmission of 
all the frames on a network on which such lost frames and 

10 time release frequently occur, a method that transmits all 

the frames not by using interframe coding but by using only 
intraframe coding, such as JPEG, has been employed. For 
example, in the case of JPEG coding, even if frame c is lost, 
decoding of the next frame can be performed. In this case, 

15 redundancy of temporal changes is not eliminated since no 

interframe coding is performed, thus causing a problem in 
that the amount of transmitted data is increased by a bad 
coding efficiency. 

[0017] In addition, there is a known technology in which, 
20 when video image data is transmitted by using interframe 

coding (interceding) and intraframe coding (intracoding) , a 
transmitting party estimates a portion of an image which may 
be affected by error, performs forcible intraframe coding on 
the estimated portion, and transmits the coded portion to a 
25 receiving party. In this case, it is difficult to estimate 
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the portion of the image which may be affected by the error. 
The estimation may be wrong, so that deterioration in image 
quality cannot completely be eliminated. 
[0018] From the above description, image coding and 
5 decoding are in demand in which, by using interframe coding 

to suppress a decrease in coding efficiency obtained when a 
small number of errors occur, even if an error occurs, a P 
frame can be decoded without awaiting an I frame which is 
later sent and image quality has less deterioration. 

10 

SUMMARY OF THE INVENTION 



[0019] In view of the above -described circumstance, the 
present invention provides an image processing apparatus and 

15 method in which, when frames of interceded video- image data 
are decoded, even if a reference frame for a frame to be 
decoded is lost, the frame to be decoded is decoded. 
[0020] According to an aspect of the present invention, 
an image processing apparatus is provided which includes an 

20 input unit for inputting image data, a first coding unit for 

coding the input image data by transforming the input image 
data into frequency components in units of blocks and coding 
the frequency components by adaptively using an intracoding 
mode and an interceding mode, a second coding unit for 

25 coding frequency components obtained by limiting the 



frequency components for reference image data for use in the 
intercoding mode, and a multiplexing unit for outputting 
combined data obtained by combining the image data coded by 
the second coding unit with the image data coded by the 
first coding unit. 

[0021] According to another aspect of the present 
invention, an image processing apparatus which includes an 
input unit for inputting image data, a first coding unit for 
coding the input image data by transforming the input image 
data into frequency components in units of blocks and coding 
the frequency components by adaptively using an intracoding 
mode and an intercoding mode, a second coding unit for 
coding frequency components obtained by limiting the 
frequency components for image data obtained by performing 
local decoding on the image data coded by the first coding 
unit , and a multiplexing unit for outputting combined data 
obtained by combining the image data coded by the second 
coding unit with the image data coded by the first coding 
unit . 

[0022] According to another aspect of the present 
invention, an image processing apparatus is provided which 
includes an input unit for inputting coded image data 
obtained by performing transformation into frequency 
components in units of blocks and coding the frequency 
components by adaptively using an intracoding mode and an 
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interceding mode, and pseudo-coded reference data obtained 
by coding frequency components obtained by limiting the 
frequency components for reference image data for use in the 
interceding mode, a first decoding unit for decoding the 
5 coded image data input by the input unit , and a second 

decoding unit for decoding the pseudo- coded reference data 
input by the input unit. When a reference image is lost in 
decoding on the image data coded in the interceding mode, 
the first decoding unit uses, as the reference image, image 
10 data obtained such that the second decoding unit decodes the 

pseudo-coded reference data. 

[0023] According to another aspect of the present 
invention, an image processing method is provided which 
includes an input step of inputting image data, a first 

15 coding step of coding the input image data by transforming 

the input image data into frequency components in units of 
blocks and coding the frequency components by adaptively 
using an intracoding mode and an interceding mode, a second 
coding step of coding frequency components obtained by 

20 limiting the frequency components for reference image data 

for use in the interceding mode, and a multiplexing step of 
outputting combined data obtained by combining the image 
data coded in the second coding step with the image data 
ceded in the first coding step. 

25 [0024] According to another aspect of the present 
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invention, an image processing method is provided which 
includes an input step of inputting image data, a first 
coding step of coding the input image data by transforming 
the input image data into frequency components in units of 
5 blocks and coding the frequency components by adapt ively 

using an intracoding mode and an interceding mode, a second 
coding step of coding frequency components obtained by 
limiting the frequency components for image data obtained by 
performing local decoding on the image data coded in the 
10 first coding step, and a multiplexing step of outputting 

combined data obtained by combining the image data coded in 
the second coding step with the image data coded in the 
first coding step. 

[0025] According to another aspect of the present 
15 invention, an image processing method is provided which 

includes an input step of inputting coded image data 
obtained by performing transformation into frequency 
components in units of blocks and coding the frequency 
components by adapt ively using an intracoding mode and an 
20 interceding mode, and pseudo-coded reference data obtained 

by coding frequency components obtained by limiting the 
frequency components for reference image data for use in the 
interceding mode, a first decoding step of decoding the 
coded image data input in the input step, and a second 
25 decoding step of decoding the pseudo-coded reference data 
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input in the input step. When a reference image is lost in 
decoding on the image data coded in the interceding mode, in 
the first decoding step, image data obtained by decoding the 
pseudo-coded reference data in the second decoding step is 
5 used as the reference image. 

[0026] Other objects, features and advantages of the 
invention will become apparent from the following detailed 
description taken in conjunction with the accompanying 
drawings . 

10 

BRIEF DESCRIPTION OF THE DRAWINGS 

[0027] Fig, 1 is an illustration of lost frames in packet 
transmission of video image data. 
15 [0028] Fig. 2 is an illustration of reference components. 

[0029] Fig. 3 is a block diagram showing an image coding 
apparatus according to a first embodiment of the present 
invention. 

[0030] Fig. 4 is a flowchart showing a process in which 
20 the image coding apparatus 2 shown in Fig. 3 perform coding 

on an I frame that does not use a reference freime. 
[0031] Fig. 5 is a flowchart showing a process in which 
the image coding apparatus 2 shown in Fig. 3 performs coding 
on a P frame. 

25 [0032] Fig. 6 is a block diagram showing an image 
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decoding apparatus according to a first embodiment of the 
present invention. 

[0033] Fig. 7 is a flowchart showing a process in which 
the image decoding apparatus 5 shown in Fig. 6 performs 
5 decoding on an I frame that does not use a reference frame. 

[0034] Fig. 8 is a flowchart showing a process in which 
the image decoding apparatus 5 shown in Fig. 6 performs 
decoding on a P frame. 

[0035] Fig. 9 is a flowchart showing a P-frame decoding 
10 process of the image decoding apparatus 5 shown in Fig. 6 in 

a case in which a normal reference frame is lost or an 
incomplete case. 

[0036] Fig. 10 is a block diagram showing a receiver in a 
first embodiment of the present invention. 
15 [0037] Fig. 11 is an illustration of error detection and 

the ON and OFF states of a control signal in response to a 
picture type. 

[0038] Fig. 12 is a flowchart showing a process in which 
the receiver 4 shown in Fig. 10 outputs a control signal. 

20 [0039] Fig. 13 is an Illustration of the structure of a 

stream in a first embodiment of the present invention. 
[0040] Fig. 14 is a block diagram showing an image 
transmission system obtained by combining the image coding 
apparatus 2 shown in Fig. 3, the image decoding apparatus 5 

25 shown in Fig. 5, and the receiver 4 shown in Fig. 10. 
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[0041] Fig. 15 Is a block diagram showing an image coding 
apparatus according to a second embodiment of the present 
invention. 

[0042] Fig. 16 is a flowchart showing a process in which 
5 the image coding apparatus 2a shown in Fig. 15 performs 

coding on an I frame that does not use a reference frame. 
[0043] Fig. 17 is a flowchart showing a process in which 
the image coding apparatus 2a performs decoding on a P frame. 
[0044] Fig. 18 is a block diagram showing the hardware 
10 structure of a computer. 

DESCRIPTION OF THE PREFERRED EMBODIMENTS 

[0045] Embodiments of the present invention are described 
15 below with reference to the accompanying drawings. 

First Embodiment 

[0046] In a first embodiment of the present invention, 
MPEG- 4 coding is described as an example of interframe 
20 coding. However, the first embodiment of the present 

invention is not limited to MPEG- 4 coding. 

[0047] An image coding apparatus 2 (Fig. 3) according to 
the first embodiment of the present invention generates 
coded data for one frame (hereinafter referred to as 
25 "pseudo- coded reference data" ) which is obtained such that 
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direct current (DC) components and some of alternating 
current (AC) components of discrete cosine transform (DCT) 
coefficients, obtained by dividing an I frame (frame which 
Is Intraframe- coded (Intra-coded) ) Into blocks and 
5 performing DCT processing on the blocks, are coded as 

effective components (AC components other than the some of 
the AC components of the DCT coefficients are regarded as 
zeroes ) . The Image coding apparatus 2 accumulates the 
pseudo-coded reference data In a memory. The Image coding 

10 apparatus 2 stores the pseudo-coded reference data In a user 

data area for a P frame (frame which Is Interf rame-coded 
(Inter-coded)) In an MPEG-4 steam. Accordingly, when a 
reference frame for a P frame is lost due to some accident 
occurring in data transmission from the image coding 

15 apparatus 2 to an image decoding apparatus, by using, as a 
reference frame, a frame obtained from the pseudo-coded 
reference data combined with stream data of the P frame, the 
P frame is decoded. 

[0048] In the first embodiment of the present invention, 
20 processing units for coding are macroblocks (hereinafter 

also referred to as "MBs"). Also, in the first embodiment, 
some of the AC components means the first to ninth 
coefficients in the case of reading the inside of a DCT 
block composed of 8 by 8 pixels in zigzag scanning order, as 
25 shown in Fig. 2. In other words, the pseudo-coded reference 
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data is coded data for one frame obtained such that DC 
components and some of the AC components as shown In Fig. 2 
are coded with the other components regarded as zeroes. 
More specifically, after DC components and some of the AC 
5 components of coefficient data, read from the DCT block, is 

coded, an End of Block (EOB) code is added to the coded 
coefficient data before coding of the DCT block ends. 
[0049] The effective components are not limited to the 
above DC components and AC components, but may be only the 
10 DC components. Alternatively, some of the mentioned above 

AC components may be those other than the nine coefficients. 
When only the DC components are used as the effective 
coefficients, the amount of pseudo-coded reference data can 
be reduced. 

15 [0050] The first embodiment of the present invention is 

specifically described below with reference to the 
accompanying drawings. 

[0051] Fig. 3 is a block diagram showing the structure of 
the image coding apparatus 2 according to the first 
20 embodiment of the present Invention. 

[0052] Referring to Fig. 3, the image coding apparatus 2 
transforms input pixel (image) data into an MPEG- 4 stream, 
which is a transmit table data string, and outputs the MPEG- 4 
stream. 

25 [0053] A memory 100 (indicated by M in Fig. 3) stores the 
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pseudo- coded reference data of an I frame. A switch 101 
selects whether the pseudo-coded reference data from the 
memory 100 is to be supplied to a multiplexer 106 (indicated 
by MUX in Fig. 3) (described later). 
5 [0054] A subtracter 102 outputs input MB data in 

unchanged form when the input MB data is an I frame. VHien 
the input MB data is a P frame, the subtracter 102 subtracts 
predictive MB data from the input MB data, and outputs the 
resultant data. 

10 [0055] A DCT unit 103 (indicated by T in Fig. 3) performs 

DCT processing on the data output by the subtracter 102. A 
quantization unit 104 (indicated by Q in Fig. 3) quantizes 
DCT coefficient data output from the DCT unit 103. 
[0056] A variable length coding unit 105 (indicated by 

15 VLC in Fig. 3) performs variable length coding on 

representative quantized values output by the quantization 
unit 104 and outputs the coded values to multiplexer 106 
( indicated by MUX in Fig . 3 ) . The variable length coding 
unit 105 generates and outputs, to the memory 100, the 

20 pseudo -coded reference data (data obtained by performing 

coding with DC components and some of the AC components used 
as effective components, and the other components regarded 
as ineffective). 

[0057] When the switch 101 is off (while interframe 
25 coding is being performed) , the multiplexer 106 directly 
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outputs a stream input from the variable length coding unit 
105. When the switch 101 is turned on^ establishing 
connection between the multiplexer 106 and the memory 100 
(while interframe coding is being performed) , the 
5 multiplexer 106 outputs a stream obtained by combining the 

pseudo-coded reference data stored in the memory 100, the 
coded data output from the variable length coding unit 105, 
and motion-vector data output by a motion compensation unit 
110 (described later) . The multiplexer 106 also combines a 
10 type code representing picture types (e.g., an I frame, a P 
frame) . 

[0058] An inverse quantization unit 107 (indicated by IQ 
in Fig. 3) inversely quantizes the representative quantized 
values from the quantization unit 104. An inverse DCT unit 

15 108 (indicated by IT in Fig. 3) performs inverse DCT 

processing on DCT coefficient data from the inverse 
quantization unit 107. An adder 109 adds the output of the 
inverse quantization unit 108 and an output from a motion 
compensation unit 110 (indicated by ME/MC in Fig. 3) 

20 (described later). The inverse quantization processing by 

the inverse quantization unit 107, and the inverse DCT 
processing by the inverse DCT unit 108 are called "local 
decoding" . 

[0059] The motion compensation unit 110 performs motion 
25 vector detection by comparing the input MB data and a frame 
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(reference frame) stored in a frame memory 111 (indicated by 
FM in Fig. 3) (described later), motion compensation in 
response to the result of the detection, and outputs 
predictive MB data. The motion compensation unit 110 also 
5 outputs motion-vector data obtained in the motion vector 

detection to the multiplexer 106. The frame memory 111 
stores image data of a reference frame used when the motion 
compensation unit 110 performs motion compensation 
processing. 

10 [0060] In the above -described configuration, in the image 

coding apparatus 2 according to the first embodiment of the 
present invention, the pseudo- coded reference data is stored 
in the memory 100, and the switch 101 is turned on when P- 
frame processing is performed, whereby the multiplexer 106 

15 outputs the combined stream obtained by combining the 

pseudo-coded reference data and the motion-vector data with 
the stream of P-frame coded data. 

[0061] Next, the operation of the image coding apparatus 
2 in Fig. 3 is fully described below. 
20 [0062] At first, an I-frame coding of the image coding 

apparatus 2 in Fig. 3 is described with reference to the 
flowchart shown in Fig. 4. 

[0063] Fig. 4 is a flowchart showing a process in which 
the image coding apparatus 2 encodes an I frame which does 
25 not use a reference frame. 
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[0064] Referring to Fig. 4, in step S200, MB data 
constituting each frame of image data is input to the image 
coding apparatus 2. In step S201, after passing through the 
subtracter 102, the input MB data is processed such that DCT 
5 processing, quantization processing, and variable length 

coding are performed by the DCT unit 103, the quantization 
unit 104, and the variable length coding unit 105, 
respectively. In step S202, a data stream which is output 
by the variable length coding unit 105 and which is obtained 
10 by variable length coding of the variable length coding unit 
105 is output from the multiplexer 106. 

[0065] In step S203, the image coding apparatus 2 stores, 
in the memory 100, the pseudo-coded reference data generated 
by the variable length coding unit 105. 

15 [0066] In step S204, image data, obtained such that the 

representative quantized values output by the quantization 
unit 104 are processed by the inverse quantization unit 107, 
the inverse DCT unit 108, and the adder 109, is stored as 
data of the next reference frame in the frame memory 111. 

20 After that, the I-freime coding process of the image coding 

apparatus 2 ends. 

[0067] Next, the operation of the image coding appsuratus 
2 in the case of coding a P frame by using a reference frame 
is described with reference to the flowchart shown in Fig. 5. 
25 [0068] Fig. 5 is a flowchart showing a process in which 
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the Image coding apparatus 2 performs P-frame coding. 
[0069] In Fig. 5, in step S300, the motion compensation 
unit 110 generates predictive MB data by referring to the 
image data (reference frame) stored in the frame memory 111. 
5 After, in step S301, input MB data to be coded is input to 

the subtracter 102, the process proceeds to step S302. 
[0070] In step S302, the subtracter 102 generates 
predictive error MB data by calculating the difference 
between the input MB data and the predictive MB data output 

10 by the motion compensation unit 110. 

[0071] In step S303, image coding on the predictive MB 
data generated in step S302 is performed by the DCT unit 103, 
the quantization unit 104, and the variable length coding 
unit 105. In step S304, the multiplexer 106 combines coded 

15 data output by the variable length coding unit 105, the 

pseudo -coded reference data (pseudo- coded reference data for 
an interface used for obtaining the predictive MB data) 
stored in the memory 100, and motion- vector data. In step 
S305, the multiplexer 106 outputs a combined stream. 

20 [0072] In step S306, the representative quantized values 

from the quantization unit 104 are processed by inverse 
quantization processing of the inverse quantization unit 107 
and inverse DCT processing of the inverse DCT unit 108. 
[0073] In step S307, the adder 109 adds the data 

25 processed in step S306 and the predictive MB data, and 
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accumulates the sum as Image data of a reference frame for 
the next frame in the frame memory 111. After that, the p- 
frame Interframe coding of the Image coding apparatus 2 ends. 
[0074] As described above. In the Image coding apparatus 
5 2 according to the first embodiment of the present Invention, 

pseudo-coded reference data of an Intraframe- coded frame (I 
frame) Is stored In the memory 100, and while an Interframe- 
coded Image (P frame) Is being processed, a stream In which 
the pseudo-coded reference data Is combined with coded data 
10 of the Interframe -coded Image (P frame) can be output. 

[0075] Next, an Image decoding apparatus 5 adapted for 
the Image coding apparatus 2 shown In Fig. 3 Is described 
below. 

[0076] Fig. 6 Is a block diagram showing the Image 
15 decoding apparatus 5 according to the first embodiment of 
the present Invention. 

[0077] In Fig. 6, the Image decoding apparatus 5 outputs 
Image data by decoding a stream Input through a network and 
a receiver (described below). 
20 [0078] A separator unit 400 (Indicated by "DE MUX" In Fig. 

6) separates the Input stream Input through the receiver 
Into a frame- type code, coded Image data, motion- vector data, 
and pseudo-coded reference data. 

[0079] A variable length decoding unit 401 (indicated by 
25 VLD in Fig. 6) performs variable length decoding on the 
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coded Image data separated by the separator unit 400, and 
outputs the resultant representative quantized values. An 
inverse quantization unit 402 (indicated by IQ in Fig. 6) 
performs inverse quantization on the representative 
quantized values output from the variable length decoding 
unit 401, and outputs the resultant DCT coefficient data. 
An inverse DCT unit 403 (indicated by IT in Fig. 6) performs 
inverse DCT processing on the DCT coefficient data output 
from the inverse quantization unit 402, and outputs the 
obtained pixel (image) data. 

[0080] An adder 404 adds the image data output from the 
inverse DCT unit 403 and predictive image data output from a 
motion compensation unit 406 (indicated by MC in Fig. 6) 
(described later). A motion vector decoding unit 405 
(indicated by "MV VLD" in Fig. 6) decodes the motion- vector 
data separated by the separator unit 400 . The motion 
compensation unit 406 generates predictive MB data from 
image data stored in a frame memory 408 or 409 (described 
later) by using the mot ion- vector data separated by the 
motion vector decoding unit 405. 

[0081] In response to an input control signal (whose 
details are described later) , a switch 407 performs 
switching about which of the frame memories 408 and 409 is 
connected to the motion compensation unit 406. The control 
signal is a signal which, when a P frame is decoded. 
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controls the switch 407 in response to a state of whether a 
reference frame for the P frame is lost or not. The control 
signal is input from a receiver (described later). In other 
words, in the image decoding apparatus 5, the switch 407 is 
controlled in response to the control signal so that, when a 
normal reference frame is not lost, the motion compensation 
unit 406 is controlled to refer to image data of the normal 
reference frame which is stored in the frame memory 408, 
while, when the normal reference frame is lost, the motion 
compensation unit 406 is controlled to refer to image data 
(image data obtained by decoding pseudo-coded reference 
data) stored in the frame memory 409. 

[0082] The frame memory 408 stores the image data output 
from the adder 404. The frame memory 409 stores image data 
obtained by decoding the pseudo-coded reference data. The 
stored image data is output by a block noise eliminating 
filter 414 (indicated by BF in Fig. 6) (described later). 
[0083] A variable length decoding unit 410 (indicated by 
VLD in Fig. 6) performs variable length decoding on the 
pseudo- coded reference data separated by the separator unit 
400, and outputs the obtained representative quantized 
values. A quantization unit 411 (indicated by IQ in Fig. 6) 
performs inverse quantization on the representative 
quantized values output from the variable length decoding 
unit 410, and outputs the obtained DCT coefficients. 
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[0084] A DCT unit 412 performs Inverse DCT processing on 
the DCT coefficients output from the quantization unit 411. 
A frame memory 413 stores the image data output from the DCT 
unit 412. The block noise eliminating filter 414 performs 
5 filtering on image data read from the frame memory 413. 

[0085] In the above -described configuration, when the 
image decoding apparatus 5 according to the first embodiment 
decodes coded data of a P frame, if a normal reference frame 
is lost, the image decoding apparatus 5 can perform decoding 

10 on the coded data of the P frame by referring to the image 
data (image data obtained by decoding the pseudo- coded 
reference data) stored in the frame memory 409. 
[0086] Next, a process of the image decoding apparatus 5 
shown in Fig. 6 is fully described below. 

15 [0087] At first, a process in which the image decoding 

apparatus 5 decodes an I frame that does not use a reference 
frame is described. 

[0088] Fig. 7 shows the process in which the image 
decoding apparatus 5 decodes an I frame that does not use a 
20 reference frame. 

[0089] In Fig. 7, in step S500, the I -frame stream output 
from the image coding apparatus 2 is input to the image 
decoding apparatus 5 through the receiver. 

[0090] In step S501, the separator unit 400 separates the 
25 input I-frame stream into a frame-type code and coded image 
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data. 

[0091] In step S502, the variable length decoding unit 
401, the inverse quantization unit 402, and the inverse DCT 
unit 403 respectively perform on the coded image data output 
5 from the separator unit 400, variable length decoding, 

inverse quantization, and inverse DCT processing, whereby 
the coded image data output from the separator unit 400 is 
decoded into image data (pixel data). At this time, the 
image data which is output as an image of reference frame 

10 for the next frame from the inverse DCT unit 403 through the 
adder 404 is stored. After that, in step S503, the image 
decoding apparatus 5 outputs the decoded I-frame pixel data. 
[0092] Next, a process in which the image decoding 
apparatus 5 decodes a P frame by using pixel data of a 

15 normal reference frame (when no reception error occurs) is 
described below. 

[0093] Fig. 8 is a flowchart showing the process of the 
image decoding apparatus 5 for decoding the P frame. 
[0094] In Fig. 8, in step S600, the P-frame streeun output 
20 from the image coding apparatus 2 is input to the image 
decoding apparatus 5 through the receiver. 

[0095] In step S601, the separator unit 400 separates the 
P-frame streeun into a frame- type code, coded image data, 
motion- vector data, and pseudo-coded reference data. 
25 [0096] In step S602, the motion vector decoding unit 405 
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decodes the motion-vector data separated by the separator 
unit 400, and outputs the decoded motion-vector data to the 
motion compensation unit 406. 

[0097] In step S603, the coded image data separated by 
5 the separator unit 400 is processed such that the variable 

length decoding unit 401, the inverse quantization unit 402, 
and the inverse DCT unit 403 perform variable length 
decoding, inverse quantization, and inverse DCT processing, 
respectively . 

10 [0098] Since a control signal from the receiver 

(described later) causes the switch 407 to select the frame 
memory 408, in step S604, the motion compensation unit 406 
outputs predictive MB data by using the motion vector 
decoded by the motion vector decoding unit 405 and referring 

15 to image data of a normal reference frame stored in the 
frame memory 408. 

[0099] In step S605, after reproducing the P frame by 
adding the image data output form the inverse DCT unit 403 
and the predictive MB data output from the motion 
20 compensation unit 406, image data of the reproduced P frame 

is stored as a reference frame for the next frame in the 
frame memory 408. 

[0100] In step S606, the image decoding apparatus 5 
outputs the reproduced P-frame image data. 
25 [0101] Next, a P-frame decoding process of the image 
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decoding apparatus 5 Is described. This P- frame decoding 
process Is performed when a P frame Is received ahead of an 
I frame to be referred to In a strecun Input through the 
receiver, and when a reference frame Is Incomplete due to 
5 some reception error occurring in the closest past stream. 

[0102] Fig. 9 Is a P-frame decoding process of the Image 
decoding apparatus 5 when a normal reference frame Is lost 
or Incomplete. 

[0103] In Fig. 9, in step S700, the P-frame stream output 
10 from the Image coding apparatus 2 is input to the image 

decoding apparatus 5. 

[0104] In step S701, the separator unit 400 separates the 
input P-frame stream into a frame- type code, coded image 
data, motion -vector data, and pseudo- coded reference data. 

15 [0105] In step S702, the image decoding apparatus 5 

stores, in the frame memory 413, decoded image data obtained 
such that the pseudo-coded reference data separated by the 
separator unit 400 is processed by the variable length 
decoding unit 410, the quantization unit 411, and the DCT 

20 unit 412. 

[0106] In step S703, the block noise eliminating filter 
414 eliminates block noise from image data read from the 
frame memory 413. 

[0107] In step S704, the image decoding apparatus 5 
25 stores, in the frame memory 409, image data output from the 
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block noise eliminating filter 414. 

[0108] In step S705^ the motion vector decoding unit 405 
decodes the motion-vector data separated by the separator 
unit 400, and outputs the decoded motion-vector data to the 
5 motion compensation unit 406. 

[0109] In step S706, the coded Image data separated by 
the separator unit 400 Is processed such that the variable 
length decoding unit 401, the Inverse quantization unit 402, 
and the Inverse DCT unit 403 perform variable length 
10 decoding. Inverse quantization, and Inverse DCT processing, 
respectively . 

[0110] Since the switch 407 selects the frame memory 409, 
based on a control signal from the receiver. In step S707, 
the motion compensation unit 406 outputs predictive MB data 

15 by using the motion-vector data decoded by the motion vector 
decoding unit 405 In step S705 and referring to the Image 
data (Image data obtained by decoding the pseudo-coded 
reference data) stored In the frame memory 409. 
[0111] In step S708, by adding the Image output from the 

20 Inverse DCT unit 403 and the predictive MB data output from 

the motion compensation unit 406, the P-frame Image data Is 
reproduced. The reproduced P-freune Image data Is stored as 
a reference frame for the next frame In the frame memory 408. 
[0112] In step S709, the Image decoding apparatus 5 

25 outputs the reproduced P-frame Image data. 
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[0113] As described above, when the Image decoding 
apparatus 5 according to the first embodiment of the present 
invention decodes a stream of interframe- coded images (P 
frames ) , if a normal reference frame is lost , the image 
5 decoding apparatus 5 can perform decoding on an interframe- 

coded image (P frame) by referring to an image (stored in 
the frame memory 409) obtained by decoding pseudo-coded 
reference data. 

[0114] Next, the above receiver that outputs a control 
10 signal and a received stream to the image decoding apparatus 

5 is described below. 

[0115] Fig. 10 is a block diagram showing the receiver 
(denoted by reference numeral 4) in the first embodiment of 
the present invention. 

15 [0116] In Fig. 10, the receiver 4 receives the streeim 

through a network and outputs the received stream to the 
image decoding apparatus 5. In this process, the receiver 4 
performs error detection, etc., on the received stream, and 
outputs a control signal to the image decoding apparatus 5. 

20 [0117] An error detecting unit 801 (denoted by ED in Fig. 

10) detects an error in the stream input through the network 
and outputs an error-detection signal. Also, the error 
detecting unit 801 controls output by skipping an error- 
detected stream to the next frame boundary. 

25 [0118] A picture- type detecting unit 802 (indicated by 
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PTD In Fig. 10) detects the picture type (code representing 
an I frame or a P frame In the first embodiment) of the 
stream externally Input through the network, and outputs a 
picture-type signal (the detected code). The stream Input 
5 to the receiver 4 Is the stream output to the network by the 

above -described Image coding apparatus 2. 

[0119] Based on the error- detection signal output from 
the error detecting unit 801 and the picture-type signal 
output from the picture-type detecting unit 802, when the 

10 reference frame for the next frame Is detected, an ON signal 

representing detection of the reference frame Is output as 
the control signal by an Identifying unit 803 (Indicated by 
J In Fig. 10), and when the reference frame for the next 
frame Is not detected, an OFF signal representing no 

15 detection of the reference frame Is output as the control 
signal by the Identifying unit 803. 

[0120] Fig. 11 shows error detection states and control 
signal switching based on a picture type. The picture type 
shown In Fig. 11 Is one for the next frame. 
20 [0121] Next, the operation of the receiver 4 shown In Fig. 

10 Is fully described below. 

[0122] Fig. 12 Is a flowchart showing a control signal 
output process of the receiver 4 shown In Fig. 10. 
[0123] In Fig. 12, In step S900, the stream Is Input to 
25 the error detecting unit 801 through the network. 
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[0124] In step S901^ the error detecting unit 801 
determines whether an error is detected in the stream. If 
the error is detected in the stream (YES in step S901), the 
error detecting unit 801 outputs an error -detect ion signal 
5 and proceeds to step S902. 

[0125] In step S902, the process skips to the start of 
the next frame. 

[0126] In step S903, the picture- type detecting unit 802 
detects the picture type of the next frame and outputs a 

10 corresponding code. 

[0127] In step S904, based on the output of the error 
detecting unit 801 and the output of the picture -type 
detecting unit 802, the identifying unit 803 outputs a 
control signal for controlling the switch 407 to be turned 

15 on/off (See Fig. 11). 

[0128] In step S905, if an error is detected, the 
receiver 4 outputs a frame- skipped stream, and if no error 
is detected, the receiver 4 directly outputs the input 
stream. 

20 [0129] Here, an example of a stream in the first 

embodiment is described below. 

[0130] Fig. 13 is an illustration of an example of a 
stream in the first embodiment. 

[0131] As shown in Fig. 13, a stream starts from a 
25 sequence start code, is followed by frame-unit data items 
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VOPs (video object planes), and ends with a sequence end 
code. Also, each VOP Includes a VOP start code at the start, 
user data, a VOP control code, and a Huffman code of Image. 
In the first embodiment of the present invention, the above 
5 pseudo-coded reference data in a reference frame for the P 

frame is included as P-frame user data. In other words, in 
a P-freune transmission block, the pseudo- coded reference 
data in the reference frame is located. Accordingly, even 
if the reference frame for the P frame is lost due to an 
10 error or the like, the P frame can be decoded by using, as a 
reference frame, a frame obtained by decoding the pseudo - 
coded reference data. 

[0132] Next, an example of an image transmitting system 
obtained by combining the image coding apparatus 2 in Fig. 3, 
15 the image decoding apparatus 5 in Fig. 6, and the receiver 4 

in Fig. 10 is described below. 

[0133] Fig. 14 is a block diagram showing the excunple of 
the image transmitting system obtained by combining the 
image coding apparatus 2 in Fig. 3, the image decoding 

20 apparatus 5 in Fig. 6, and the receiver 4 in Fig. 10. 

[0134] In Fig. 14, a camera 1 captures an image and 
outputs image data. The image coding apparatus 2 outputs a 
stream based on the image data output from the camera 1. A 
transmitter 3 transmits the stream output by the image 

25 encoding apparatus 2 to a network. The receiver 4 receives 
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the stream transmitted by the transmitter 3 through the 
network. The image decoding apparatus 5 decodes image data 
from the streeim received by the receiver 4 . A monitor 6 
displays an image based on the image data decoded by the 
5 image decoding apparatus 5. 

[0135] In the first embodiment of the present invention, 
the MPEG- 4 standard is used as interframe coding on image 
data. However, the interframe coding is not limited to the 
MPEG- 4 standard, but other interframe coding may be used. 

10 Although, in the first embodiment of the present invention, 
an error detecting function and a picture-type detecting 
function are provided in the receiver 4 , another 
configuration may be used, such as providing the functions 
in the image decoding apparatus 5. In the first embodiment 

15 of the present invention, block-noise eliminating filtering 
is performed on a frame in which pseudo-coded reference data 
is decoded. However, for reducing the processing load, a 
structure that does not use the block-noise eliminating 
filtering may be used when the quality of the decoded image 

20 is in an allowable range. 

[0136] As is clear from the above description, in the 
case of decoding a stream of frames on which interframe - 
coding is performed, in the image coding apparatus 2 and 
image decoding apparatus 5 according to the first embodiment 

25 of the present invention, even if a reference frame in the 
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frame is lost, decoding can be performed by decoding pseudo- 
coded reference data In combined form and using the decoded 
pseudo-coded reference data as a reference frame, whereby a 
high reproducible Image can be obtained. 
5 [0137] In addition, in a stream configuration in the 

first embodiment of the present Invention, by locating 
pseudo- coded reference data of an I frame in the user data, 
a known decoding apparatus which does not have the decoding 
function can perform known decoding (image quality 
10 deteriorates when error occurs). 



Second Embodiment 

[0138] Next, an image coding apparatus according to a 
second embodiment of the present invention is described 

15 below. 

[0139] Fig. 15 is a block diagram showing the 
configuration of an image coding apparatus 2a according to 
the second embodiment of the present Invention. 
[0140] The image coding apparatus 2a shown in Fig. 15 

20 differs from the above image encoding apparatus 2 in that DC 

components and some of the AC components of a reference 
frame stored in locally- decoded form in the freune memory 111 
are coded as effective components. In other words, although 
the first embodiment of the present invention generates 

25 pseudo-coded reference data from only an I frame, the second 
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embodiment of the present Invention can also generate 
pseudo-coded reference data from a P frame. 
[0141] Differences of the Image coding apparatus 2a are 
mainly described below* 
5 [0142] As shovm In Fig. 15, a description of functional 

units 101 to 111 of the Image coding apparatus 2a Is omitted 
since they are Identical to those of the Image coding 
apparatus 2 described above. 

[0143] In Fig. 15, a DCT unit 10 (Indicated by T In Fig. 

10 15) performs DCT processing on Image data of a reference 

frame which Is stored In the frame memory 111. A generating 
unit 11 (Indicated by G In Fig. 15) outputs coefficient data 
In which components other than the DC components and some of 
the AC components (see Fig. 2) In the coefficient data 

15 obtained by the DCT unit 10 are regarded as zeroes. 

[0144] The effective components are not limited to the DC 
components and some of the AC components, but only the DC 
components may be regarded as effective and the other 
components may be regarded as zeroes. Alternatively, some 

20 of the AC components may be coefficients other the nine 
coefficients shown In Fig. 2. 

[0145] A quantization unit 12 (Indicated by Q In Fig. 15) 
quantizes the coefficient data output from the generating 
unit 11, and outputs representative quantized values. A 
25 variable length coding unit 13 (Indicated by VLC In Fig. 15) 
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performs variable length coding on the representative 
quantized values . 

[0146] In the above -described configuration, the Image 
coding apparatus 2a according to the second embodiment of 
5 the present Invention performs quantization and variable 

length coding by using, as effective data, DC components and 
some of the AC components of a reference frame on which 
local decoding is performed, whereby pseudo- coded reference 
data in the reference frame is generated. Also, when a P 
10 frame is processed, by switching on the switch 101, the 

image coding apparatus 2a uses the multiplexer 106 to output 
a stream in which the pseudo -coded reference data is 
combined with P- frame coded data. 

[0147] Next, the operation of the image coding apparatus 

15 2a in Fig. 15 is fully described below. 

[0148] At first, an I-frame coding process of the image 
coding apparatus 2a, that is, a process for coding an I 
frame which does not use a reference frame, is described 
below with reference to the flowchart shown in Fig. 16. 

20 [0149] Fig. 16 is a flowchart showing the I-frame (using 

no reference frame) coding process of the image coding 
apparatus 2a. 

[0150] In the process of the image coding apparatus 2a 
shown in Fig. 16, a description of steps denoted by 
25 reference numerals identical to those in the case of the 
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image coding apparatus 2 according to the first embodiment 
is simplified since the steps are similar to those in Fig. 4. 
[0151] As shown in Fig. 16. in steps S200 to 202, the 
image coding apparatus 2a outputs a streeim by performing 
5 image coding process on input MB data. In step S204a, the 

frame memory 111 stores, as a reference image (reference 
frame) for the next frame, image data obtained such that 
representative quantized values output by the quantization 
unit 104 are processed by the inverse quantization unit 107, 
10 the inverse DCT unit 108, and the adder 109. After that, 

the I -frame coding process of the image coding apparatus 2a 
ends . 

[0152] Next, a process in which the image coding 
apparatus 2a uses a reference frame to code a P frame is 
15 described below with reference to the flowchart shown in Fig. 

17. 

[0153] Fig. 17 is a flowchart showing the process in 
which the image coding apparatus 2a (in Fig. 15) uses a 
reference frame to code a P frame. In the process shown in 

20 Fig. 17 of the image coding apparatus 2a, a description of 

steps denoted by reference numerals identical to those in 
the case of the image coding apparatus 2 (in Fig. 5) 
according to the first embodiment is simplified since the 
steps are similar to those in Fig. 5. 

25 [0154] As shown in Fig. 17, in steps S300 to S303, the 
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Image coding apparatus 2a generates predictive MB data for 
coding the P frame. Based on the generated predictive MB 
data, the image coding apparatus 2a generates predictive 
error MB data, and performs coding on the generated 
5 predictive error MB data. 

[0155] In step S304a, the DCT unit 10 transforms the 
image data (reference frame) stored in the frame memory 111 
into DCT coefficient data. Next, the generating unit 11 
generates coefficient data in which, in the DCT coefficient 
10 data, DC components and some of the AC components are used 

as effective components (the other components are regarded 
as zeroes) . 

[0156] In step S304b, pseudo-coded reference data of a 
reference frame is generated such that the coefficient data 

15 generated by the generating unit 12 is coded by the 

quantization unit 12 and the variable length coding unit 13. 
[0157] In step S304C, the multiplexer 106 outputs a 
stream in which the pseudo-coded reference data output from 
the variable length coding unit 13 and the predictive MB 

20 data output from the variable length coding unit 105 are 

combined with each other. The subsequent steps S305 to S307 
are identical to those in Fig. 5. 

[0158] As described above, when an interframe- coded image 
(P frcime) is processed, the image coding apparatus 2a can 
25 generate pseudo- coded reference data from the reference 
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frame stored in the frame memory 111, and can output a 
stream in which the pseudo-coded reference data is combined 
with the interframe- coded image (P frame). 
[0159] Decoding of the image data coded by the image 
coding apparatus 2a in Fig. 15 may be performed simileirly to 
the first embodiment of the present invention by using the 
image decoding apparatus 5 in Fig. 6. 

[0160] As is clear from the above description, when a 
stream output by the image coding apparatus 2a according to 
the second embodiment of the present invention is decoded by 
the image decoding apparatus 5, even if a reference frame 
for a P frame is lost, pseudo-coded reference data in 
combined form is decoded and used as a reference image, thus 
enabling decoding on the P frame, whereby a high 
reproducible image can be obtained. In addition, since the 
stream configuration in the second embodiment is similar to 
that shown in Fig. 13, by locating pseudo-coded reference 
data as user data, a decoding apparatus of the related art 
can perform known decoding (image quality deteriorates when 
error occurs ) . 

Other Embodiments 

[0161] Embodiments of the present invention in the case 
of using a computer to realize the above image coding 
apparatuses and image decoding apparatuses are described 
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below . 

[0162] Fig. 18 is a block diagram showing the 
configuration of hardware of a computer. 

[0163] In Fig. 18, a central processing unit (CPU) 1000 
5 controls the entirety of the computer and performs various 

processes. A memory 1001 provides the operating system (OS) 
required for controlling the computer, software, data, and 
the storage area required for arithmetic operations. The 
memory 1001 is also used as a work area when the CPU 1000 
10 performs various processes. The above software includes 
program code for allowing the CPU 1000 to execute the 
processes in accordance with the flowcharts shown in Figs. 4, 
5, 16, and 17. 

[0164] A bus 1002 connects various devices in the 
15 computer and exchanges data and control signals. Recording 
media 1003 is a type of recording medixim for storing various 
types of software. Recording media 1004 is a type of 
storage media for storing image data. A monitor 1005 
displays images, system messages from the computer, etc. A 
20 communication interface 1007 transmits coded data (stream) 

to a communication line 1008. The communication line 1008 
is connected to a local area network outside the computer, a 
public circuit, a wireless link, broadcasting radio waves, 
etc. A terminal 1006 activates the computer and sets up 
25 various conditions such as a bit rate. The memory 1001 
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further Includes an area for reading Image data, a code area 
for temporarily storing coded data, and a working area for 
storing parameters, etc., for various arithmetic operations. 
[0165] In this configuration, before performing 
5 processing, by using the terminal 1006, from video data 

stored in the recording media 1004, video data to be coded 
Is selected and activation of the computer Is commanded. 
Then, software stored In the recording media 1003 is loaded 
into the memory 1001 and is started. The CPU 1000 executes 

10 program code in accordance with the flowcharts shown in Figs. 

4, 5, 16, and 17, whereby a video data coding operation is 
realized. In other words, by using the CPU 1000 and the 
memory 1001 to constitute the code of each processing block 
in the image coding apparatus 2 or 2a shown in Fig. 3 or 15, 

15 and loading, into the memory 1001, a program for 

implementing the function of the processing block, the 
function is realized. 

[0166] As described above, the computer shown in Fig. 18 
realizes the image coding apparatus 2 according to the first 
20 embodiment of the present invention and the image coding 

apparatus 2a according to the second embodiment of the 
present invention. 

[0167] Also, it is obvious to realize the image decoding 
apparatus 5 by storing program code in accordance with the 
25 flowcharts shown in Figs. 7 to 9 . In other words, by using 
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the CPU 1000 and the memory 1001 to constitute the code of 
each processing block In the Image decoding apparatus 5 
shown in Fig. 6, and loading, into the memory 1001, a 
program for implementing the function of the processing 
5 block, the function is realized. 

[0168] The computer shown in Fig. 18 may include 
dedicated hardware having a function similar to part of the 
processing blocks shown in Figs. 3, 6, and 15. The memory 
1001 is formed by a hard disk drive or magneto -optical disk 

10 drive, a nonvolatile memory such as a frame memory, a read- 
only memory such as a compact-disk read-only memory (CD-ROM), 
a volatile memory such as a random access memory (RAM), or a 
computer-readable, computer-writable recording medium which 
is a combination of the above types of memories. 

15 [0169] The "computer -readable recording medium" means a 

portable medium such as a flexible disk, a magneto -optical 
disk, a ROM, or a CD-ROM, or a storage device such as a hard 
disk which is loaded into a computer system. Also, the 
"computer-readable recording medium" includes a type of 

20 memory that holds program code for a predetermined time, 

such as a volatile memory (RAM) in a computer system that is 
used as a server or a client when the program code is 
transmitted through a network, such as the Internet, or a 
communication line such as a telephone line. 

25 [0170] In addition, the program code may be transmitted 
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from a computer system in which the program code is stored 
in a storage device or the like, to another computer system 
through a transmission medium or by transmission waves in 
the transmission medium. The "transmission medium" that 
5 transmits the program means a medium having an information 

transmitting function such as a network (communication 
network), such as the Internet, and a communication link 
(communication line) such as a telephone link. 
[0171] In addition, the above program code may be for 
10 realizing part of the above functions. Moreover, the above 
program may be what is realized in combination with program 
code already recorded in a computer system, for example, a 
so-called "difference file". 

[0172] Moreover, also a program product, such as a 
15 computer -readable recording medium with the above program 
code recorded therein, can be applied as an embodiment of 
the present invention. The above program code, the 
recording medium, the transmission medium, and the program 
product are included in the present invention. 
20 [0173] In other words, the foregoing description of 

embodiments has been given for illustrative purposes only 
and not to be construed as imposing any limitation in every 
respect . 

[0174] The scope of the invention is, therefore, to be 
25 determined solely by the following claims and not limited by 
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the text of the specifications and alterations made within a 
scope equivalent to the scope of the claims fall within the 
true spirit and scope of the invention. 



